Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.16.4
Master_User: replica
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: MASTERMYSQL01-bin.000009
Read_Master_Log_Pos: 4

Relay_Log_Pos: 3630
Relay_Master_Log_File: MASTERMYSQL01-bin.000009
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 4
Relay_Log_Space: 3630
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 1519187

1 row in set (0.00 sec)

Above highlighted rows must be indicate related log files and  Slave_IO_Running and   Slave_SQL_Running: must be to YES.

Step 6:

On master 1:

mysql> show master status;
+————————+———-+————–+——————+
| File                   | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+————————+———-+————–+——————+
|MysqlMYSQL01-bin.000008 |      410 | adam         |                  |
+————————+———-+————–+——————+
1 row in set (0.00 sec)

The above scenario is for master-slave, now we will create a slave master scenario for the same systems and it will work as master master.

Step 7:

On Master2/Slave 1, edit my.cnf and master entries into it:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
server-id=2

master-host = 192.168.16.4
master-user = replication
master-password = slave
master-port = 3306

log-bin                     #information for becoming master added
binlog-do-db=adam

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Step 8:

Create a replication slave account on master2 for master1:

mysql> grant replication slave on *.* to ‘replication’@192.168.16.4 identified by ’slave2′;

Step 9:

Edit my.cnf on master1 for information of its master.

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

log-bin
binlog-do-db=adam
binlog-ignore-db=mysql
binlog-ignore-db=test

server-id=1
#information for becoming slave.
master-host = 192.168.16.5
master-user = replication
master-password = slave2
master-port = 3306

[mysql.server]user=mysqlbasedir=/var/lib

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Step 10:

Restart both mysql master1 and master2.

On mysql master1:

mysql> start slave;

On mysql master2:

mysql > show master status;

On mysql master 1:

mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.16.5
Master_User: replica
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: Mysql1MYSQL02-bin.000008
Read_Master_Log_Pos: 410
Relay_Log_File: Mysql1MYSQL01-relay-bin.000008
Relay_Log_Pos: 445
Relay_Master_Log_File: Mysql1MYSQL02-bin.000008
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 410
Relay_Log_Space: 445
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 103799
1 row in set (0.00 sec)

ERROR:
No query specified

Check for the hightlighted rows, make sure its running. Now you can create tables in the database and you will see changes in slave. Enjoy!!

[client]


port		= 3306
socket		= /var/run/mysqld/mysqld.sock


[mysqld_safe]


socket		= /var/run/mysqld/mysqld.sock
nice		= 0


[mysqld]


user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
language	= /usr/share/mysql/english
skip-external-locking
bind-address		= 192.168.2.200
key_buffer		= 16M
max_allowed_packet	= 16000M
thread_stack		= 128K
thread_cache_size	= 8
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
query_cache_limit       = 1M
query_cache_size        = 1024M
innodb_buffer_pool_size=2048M
innodb_additional_mem_pool_size=400M
innodb_log_file_size=25M
innodb_log_buffer_size=80M
sort_buffer=512M
thread_concurrency=32
record_buffer=128M
query_cache_type=2
tmp_table_size=1G
#log		= /var/log/mysql/mysql.log
#log_slow_queries	= /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#server-id		= 1
#log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size         = 100M
#binlog_do_db		= include_database_name
#binlog_ignore_db	= include_database_name
#skip-innodb
# chroot = /var/lib/mysql/
[mysqldump]
quick
quote-names
max_allowed_packet	= 16M


[mysql]


#no-auto-rehash	# faster start of mysql but no tab completition


[isamchk]


key_buffer		= 16M


* Turn off HyperThreading -- it tends to slow things down.


* key_buffer = 16M -- very small if you use any MyISAM. In which case, set it to 3000M. (It must not be bigger than 4G.) Since you are not using any MyISAM, leave it at 16M.
* query_cache_size = 1024M -- The bigger it is, the slower it runs. Suggest 200M.
* innodb_buffer_pool_size=2048M -- You have 16GB of RAM. Is it 'all' available to MySQL? If so, and you are using InnoDB exclusively, suggest 13G.
* If mixture of MyISAM and Innodb, adjust proportionally.
* InnoDB database with 1,7 GiB of data? Wasted your money on so much RAM.
* #log_slow_queries	= /var/log/mysql/mysql-slow.log
#long_query_time = 2
Do you have any control over the schema or queries? Perhaps not if you are using a canned product. If you did have control, I would recommend turning on the slowlog and using it to help tune things.
* #max_connections = 100 -- I don't know what the default is; keep an eye on max_used_connections. If you run out of connections set max_connections appropriately.
* #table_cache = 64 -- You can probably set this to something, then watch open_tables to see whether it needs to be bigger.
Executive summary: Your current settings are close to the best available.

This is one of the Best Configuration file from MYSQL

ADMINISTRATION BIBLE BOOK.

The configuration file supports options for multiple programs. Each program has its own directive,
which is a keyword in square brackets, such as [mysqld] or [client]. Here is a sample
configuration file from a production system:
# The following options will be passed to all MySQL clients
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
#this will prevent mysql from starting
port = 3306
socket = /var/lib/mysql/mysql.sock
datadir =/var/lib/mysql
log_slow_queries = /var/log/mysql/mysqld_slow-queries.log
long_query_time = 4
max_connections=200
max_connect_errors = 400
wait_timeout=7200
connect_timeout=10
key_buffer = 512M
tmp_table_size = 32M
max_heap_table_size = 32M
max_allowed_packet = 32M
table_cache = 1800
join_buffer_size = 8M
sort_buffer_size = 16M
read_buffer_size = 8M
read_rnd_buffer_size = 524288
myisam_sort_buffer_size = 256M
thread_cache_size = 384
bulk_insert_buffer_size = 8M
query_cache_limit = 4M
query_cache_size = 128M
query_cache_type = 1
362
MySQL Server Tuning 10
query_prealloc_size = 65536
query_alloc_block_size = 131072
# Try number of CPU’s*2 for thread_concurrency
thread_concurrency = 4
#innodb configuration
innodb_data_home_dir = /mysql/data
innodb_data_file_path=ibdata1:200M:autoextend
innodb_log_group_home_dir = /mysql/data/data
innodb_log_arch_dir = /mysql/data/data
innodb_buffer_pool_size = 24576M
innodb_additional_mem_pool_size = 32M
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 120
sync-binlog = 1
innodb_support_xa = 0
innodb_thread_concurrency = 128
innodb_file_per_table
# binary logging is required for replication
log-bin=mysql-bin
max_binlog_size = 1024M
server-id = 4
slave-skip-errors = 1062
expire-logs-days = 7
[mysqldump]
quick
max_allowed_packet = 16M
[mysql.server]
user=mysql
group=mysql
basedir=/var/lib
[mysqld_safe]
nice = -5
open_files_limit = 8192
log-error=/var/log/mysql/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
The configuration file supports options for multiple programs. Each program has its own directive,which is a keyword in square brackets, such as [mysqld] or [client]. Here is a sampleconfiguration file from a production system:# The following options will be passed to all MySQL clients[client]port = 3306socket = /var/lib/mysql/mysql.sock[mysqld]#this will prevent mysql from startingport = 3306socket = /var/lib/mysql/mysql.sockdatadir =/var/lib/mysqllog_slow_queries = /var/log/mysql/mysqld_slow-queries.loglong_query_time = 4max_connections=200max_connect_errors = 400wait_timeout=7200connect_timeout=10key_buffer = 512Mtmp_table_size = 32Mmax_heap_table_size = 32Mmax_allowed_packet = 32Mtable_cache = 1800join_buffer_size = 8Msort_buffer_size = 16Mread_buffer_size = 8Mread_rnd_buffer_size = 524288myisam_sort_buffer_size = 256Mthread_cache_size = 384bulk_insert_buffer_size = 8Mquery_cache_limit = 4Mquery_cache_size = 128Mquery_cache_type = 1362MySQL Server Tuning 10query_prealloc_size = 65536query_alloc_block_size = 131072# Try number of CPU’s*2 for thread_concurrencythread_concurrency = 4#innodb configurationinnodb_data_home_dir = /mysql/datainnodb_data_file_path=ibdata1:200M:autoextendinnodb_log_group_home_dir = /mysql/data/datainnodb_log_arch_dir = /mysql/data/datainnodb_buffer_pool_size = 24576Minnodb_additional_mem_pool_size = 32Minnodb_log_file_size = 1024Minnodb_log_files_in_group = 2innodb_log_buffer_size = 16Minnodb_flush_log_at_trx_commit = 1innodb_lock_wait_timeout = 120sync-binlog = 1innodb_support_xa = 0innodb_thread_concurrency = 128innodb_file_per_table# binary logging is required for replicationlog-bin=mysql-binmax_binlog_size = 1024Mserver-id = 4slave-skip-errors = 1062expire-logs-days = 7[mysqldump]quickmax_allowed_packet = 16M[mysql.server]user=mysqlgroup=mysqlbasedir=/var/lib[mysqld_safe]nice = -5open_files_limit = 8192log-error=/var/log/mysql/mysqld.logpid-file=/var/run/mysqld/mysqld.pid

Are you seeing a MySQL error that says InnoDB support isn’t enabled, even though it is? This article explains why it happens and how to fix it.

The symptom

Suppose you call SHOW INNODB STATUS or another InnoDB-specific command and MySQL reports the following error:

“ERROR 1235 (42000): Cannot call SHOW INNODB STATUS because skip-innodb is defined”

Yet you search the MySQL configuration files and find that’s not true. And you know you have InnoDB tables, too. What’s going on?

As it turns out, the error message is a bit misleading. Many problems will cause this error message.

Dig deeper

If InnoDB says it’s disabled, it probably is. Look at a couple other things. Does SHOW ENGINES report InnoDB is disabled? How about SHOW VARIABLES LIKE 'have_innodb'? Try SHOW TABLE STATUS on an InnoDB table — are most columns NULL?

If so, you most likely have an InnoDB configuration error. Not that you’ve disabled it with skip-innodb, but there’s something wrong. If so, MySQL will still start, but the InnoDB storage engine, and tables that use it, will be disabled.

One possible solution

When this happened to me, it was a config file upgrade that I didn’t check carefully. The old directive for the InnoDB data file was as follows:

innodb_data_file_path           = ibdata1:10M:autoextend

When I upgraded the file, I changed it to

innodb_data_file_path = ibdata1:10M:autoextend:max:128M

That wouldn’t have been a problem, except the file was already larger than 128MB. This is a slightly hard error to catch sometimes, because it may not show up in your MySQL error log (it doesn’t on my Ubuntu laptop when I deliberately force the error to happen).

A note of general caution

For those of you who are new to InnoDB configuration and administration, be careful. InnoDB has to be told exactly what to do. If you do anything wrong, such as set the permissions wrong on InnoDB’s log or data files or directories, change the file sizes, or any of a bunch of other mistakes, it will be very unforgiving. It may even wipe your existing log and data files and replace them with new ones full of zeroes (yes, this will delete all your data).

The MySQL error log is your friend, but in many cases InnoDB doesn’t flush any output to it for a long time, so you might for example start MySQL and see “MySQL NOT started.” It might just be that InnoDB wasn’t shut down nicely and has to roll back transactions to get to a consistent state. If so, that information will show up in the log files, but it might take a Very Long Time.

If you need help

You can get help on the #mysql IRC channel, mailing lists, or just read the MySQL manual. I like the IRC channel best myself. It’s friendly and there are a lot of smart people there to answer your questions.

[mysqld]

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-innodb    # (if you don’t want to use INNODB Tables)
max_connections = 500
key_buffer = 16M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 64
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1

[mysqld_safe]

err-log=/var/log/mysqld.log
open_files_limit = 8192

[mysqldump]

quick
max_allowed_packet = 16M

[myisamchk]

key_buffer = 64M

sort_buffer = 64M

read_buffer = 16M

write_buffer = 16M

[mysql.server]

user=mysql

How To Set Up Database Replication In MySQL

Version 1.1
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited: 01/14/2006

This tutorial describes how to set up database replication in MySQL. MySQL replication allows you to have an exact copy of a database from a master server on another server (slave), and all updates to the database on the master server are immediately replicated to the database on the slave server so that both databases are in sync. This is not a backup policy because an accidentally issued DELETE command will also be carried out on the slave; but replication can help protect against hardware failures though.

In this tutorial I will show how to replicate the database exampledb from the master with the IP address 192.168.0.100 to a slave. Both systems (master and slave) are running Debian Sarge; however, the configuration should apply to almost all distributions with little or no modification.

Both systems have MySQL installed, and the database exampledb with tables and data is already existing on the master, but not on the slave.

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

1 Configure The Master

First we have to edit /etc/mysql/my.cnf. We have to enable networking for MySQL, and MySQL should listen on all IP addresses, therefore we comment out these lines (if existant):

#skip-networking
#bind-address            = 127.0.0.1

Furthermore we have to tell MySQL for which database it should write logs (these logs are used by the slave to see what has changed on the master), which log file it should use, and we have to specify that this MySQL server is the master. We want to replicate the database exampledb, so we put the following lines into /etc/mysql/my.cnf:

log-bin = /var/log/mysql/mysql-bin.log
binlog-do-db=exampledb
server-id=1

Then we restart MySQL:

/etc/init.d/mysql restart

Then we log into the MySQL database as root and create a user with replication privileges:

mysql -u root -p
Enter password:

Now we are on the MySQL shell.

GRANT REPLICATION SLAVE ON *.* TO ’slave_user’@'%’ IDENTIFIED BY ‘<some_password>’; (Replace <some_password> with a real password!)
FLUSH PRIVILEGES;

Next (still on the MySQL shell) do this:

USE exampledb;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

The last command will show something like this:

+---------------+----------+--------------+------------------+
| File          | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.006 | 183      | exampledb    |                  |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Write down this information, we will need it later on the slave!

Then leave the MySQL shell:

quit;

If you want to follow the first method, then do this:

mysqldump -u root -p<password> –opt exampledb > exampledb.sql (Replace <password> with the real password for the MySQL user root! Important: There is nospace between -p and <password>!)

This will create an SQL dump of exampledb in the file exampledb.sql. Transfer this file to your slave server!

If you want to go the LOAD DATA FROM MASTER; way then there is nothing you must do right now.

mysql -u root -p
Enter password:
UNLOCK TABLES;
quit;

Now the configuration on the master is finished. On to the slave…

2 Configure The Slave

On the slave we first have to create the database exampledb:

mysql -u root -p
Enter password:
CREATE DATABASE exampledb;
quit;

mysql -u root -p<password> exampledb < /path/to/exampledb.sql (Replace <password> with the real password for the MySQL user root! Important: There is no space between -p and <password>!)

If you want to go the LOAD DATA FROM MASTER; way then there is nothing you must do right now.

server-id=2
master-host=192.168.0.100
master-user=slave_user
master-password=secret
master-connect-retry=60
replicate-do-db=exampledb

Then we restart MySQL:

/etc/init.d/mysql restart

mysql -u root -p
Enter password:
LOAD DATA FROM MASTER;
quit;

If you have phpMyAdmin installed on the slave you can now check if all tables/data from the master exampledb is also available on the slave exampledb.

mysql -u root -p
Enter password:
SLAVE STOP;

In the next command (still on the MySQL shell) you have to replace the values appropriately:

CHANGE MASTER TO MASTER_HOST=’192.168.0.100′, MASTER_USER=’slave_user’, MASTER_PASSWORD=’<some_password>‘, MASTER_LOG_FILE=’mysql-bin.006′, MASTER_LOG_POS=183;

• MASTER_HOST is the IP address or hostname of the master (in this example it is 192.168.0.100).
• MASTER_USER is the user we granted replication privileges on the master.
• MASTER_PASSWORD is the password of MASTER_USER on the master.
• MASTER_LOG_FILE is the file MySQL gave back when you ran SHOW MASTER STATUS; on the master.
• MASTER_LOG_POS is the position MySQL gave back when you ran SHOW MASTER STATUS; on the master.

Now all that is left to do is start the slave. Still on the MySQL shell we run

START SLAVE;
quit;

That’s it! Now whenever exampledb is updated on the master, all changes will be replicated to exampledb on the slave. Test it!

MY.CNF Configuration file

The configuration file supports options for multiple programs. Each program has its own directive, which is a keyword in square brackets, such as [mysqld] or [client]. Here is a sample configuration file from a production system:

# The following options will be passed to all MySQL clients

[client]

port = 3306

socket = /var/lib/mysql/mysql.sock

[mysqld]

#this will prevent mysql from starting

port = 3306

socket = /var/lib/mysql/mysql.sock

datadir =/var/lib/mysql

log_slow_queries = /var/log/mysql/mysqld_slow-queries.log

long_query_time = 4

max_connections=200

max_connect_errors = 400

wait_timeout=7200

connect_timeout=10

key_buffer = 512M

tmp_table_size = 32M

max_heap_table_size = 32M

max_allowed_packet = 32M

table_cache = 1800

join_buffer_size = 8M

sort_buffer_size = 16M

read_buffer_size = 8M

read_rnd_buffer_size = 524288

myisam_sort_buffer_size = 256M

thread_cache_size = 384

bulk_insert_buffer_size = 8M

query_cache_limit = 4M

query_cache_size = 128M

query_cache_type = 1

362

MySQL Server Tuning 10

query_prealloc_size = 65536

query_alloc_block_size = 131072

# Try number of CPU’s*2 for thread_concurrency

thread_concurrency = 4

#innodb configuration

innodb_data_home_dir = /mysql/data

innodb_data_file_path=ibdata1:200M:autoextend

innodb_log_group_home_dir = /mysql/data/data

innodb_log_arch_dir = /mysql/data/data

innodb_buffer_pool_size = 24576M

innodb_additional_mem_pool_size = 32M

innodb_log_file_size = 1024M

innodb_log_files_in_group = 2

innodb_log_buffer_size = 16M

innodb_flush_log_at_trx_commit = 1

innodb_lock_wait_timeout = 120

sync-binlog = 1

innodb_support_xa = 0

innodb_thread_concurrency = 128

innodb_file_per_table

# binary logging is required for replication

log-bin=mysql-bin

max_binlog_size = 1024M

server-id = 4

slave-skip-errors = 1062

expire-logs-days = 7

[mysqldump]

quick

max_allowed_packet = 16M

[mysql.server]

user=mysql

group=mysql

basedir=/var/lib

[mysqld_safe]

nice = -5

open_files_limit = 8192

log-error=/var/log/mysql/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

RAID
(Redundant Array of Independent Disks)

Raid 0 (zero): Only One Copy of Data spread out into Multiple Disks. if One copy Lost all the Data is Lost.

Raid 1 : Two Disks Mirrored if one fails Other will Pickup.

Raid 5: Combines Three of more Disks.  Storage Capacity is Reduced by One Disk.

Raid 10: Combination of Raid 1 and 0 Storage Capacity – 50% Mirrored.

Linux:

LVM: Logical Volume Manager: Which will Separate Log Files on a Different Server.

File System

EXT 2: Good Perfoermance & Slower Crash Recovery.

EXT 3:  Good Performance over EXT2 Filer System.

IBM JFS:  Fast Recovery After a file System Failure.

Tuning MYSQL Server

Show Innodb Status: Which will show all the Status Information of Tables, Queries, Locks, Memory Usage, Memory Dumps, Cache, Flushed Transactions, Thread information Etc.

SHOW VARIABLES: Displays all variables and the usage    ex: Show Global Variables Like ‘%tmp%/’

By default, SHOW STATUS will show SESSION variables. This may or may not be your intention, so to avoid confusion, always specify GLOBAL or SESSION as a keyword.

mysql> SHOW GLOBAL STATUS LIKE ’%tmp%’;

+————————-+———+

| Variable_name | Value |

+————————-+———+

| Created_tmp_disk_tables | 2744623 |

| Created_tmp_files | 147344 |

| Created_tmp_tables | 5003238 |

+————————-+———+

The SHOW VARIABLES can be used to determine the values of system variables. This is a very useful command when tuning a server. As with the SHOW STATUS command, it can use the LIKE modifier, and you should always specify GLOBAL or SESSION to avoid possible confusion.

If you wanted to see the global settings for temporary tables:

mysql> SHOW GLOBAL VARIABLES LIKE ’%tmp\_%’;

+——————-+———-+

| Variable_name | Value |

+——————-+———-+

| max_tmp_tables | 32 |

| tmp_table_size | 33554432 |

+——————-+———-+

“max_tmp_tables”,”32″

“slave_load_tmpdir”,”/tmp”

“tmp_table_size”,”16777216″

“tmpdir”,”/tmp”

INNODB Performance Tuning  Parameters

INNODB_BUFFER_POOL_SIZE : 70-80% memory can be allocated to Buffer Pool size, Set it to 12gb on a 16gb box.

INNODB_LOG_FILE_SIZE:  This Depends on your Recovery Speed but 256mb Seems to be a good balance between Reasonable Recovery and Good Time Performance.

INNODB_LOG_BUFFER_SIZE: This can be Set to the

INNODB_FLUSH_LOG_AT_TRX_COMMAND: Works good before Server Crashes.

INNODB_THREAD_CONCURRENCY: 8

INNODB_FLUSH_METHOD: O-Direct

INNODB_FILE_PER_TABLE:

Transaction Isolation: Read Committed: This Option has some performance Benefits especially in locking in 5.0 and Even more to come with MySQL 5.1 and Row Level Application

Application Tuning for INNODB:

ERROR log:  Sql Daemon Start Stop, Critical Errors will be Logged in this Error Log File

Binary Log: Update, Delete, Insert,  Statements will be logged to Binary Log file.

General Query Log:  All Select Queries, general data retrieval queries will be recorded.

Slow Query Log:  Queries which are running slow and can be logged in to the file.

To activate the query log, simply place:
log-slow-queries = [slow_query_log_filename]
in your configuration file (my.cnf or my.ini), slow_query_log_filename being the optional filename for your log file. If you don’t supply a filename, the default name will be used, which is the name of the host machine, with -slow.log being appended.

The slow query log logs all queries that take longer than long_query_time, which is usually 10 seconds by default (more than long enough for a self-respecting query to complete). You can alter the long_query_time in the configuration file. The following example sets the time to 5 seconds:
set-variable = long_query_time = 5
The slow query log can also optionally log all queries that don’t use an index by placing the following in the configuration file:

log-long-format

s of MySQL 5.1.6, the destination can be a file or a table, or both
mysqld with the --log-slow-queries[=file_name] option to enable the slow query log, and optionally use --log-output to specify the log destination (as described in Section 5.2.1, “Selecting General Query and Slow Query Log Output Destinations”).

Tee:  Turns on MYSQL Logging  to a specified file. ex: Tee / Filename,
NoTee:  Will stop logging mysql.

Memory Storage Engine Configuration

max_heap_table_size: The Maximum size of memory tables is limited by the system variable, it has a default value of 16 mega bytes.

Init_File: This can be used to specify a file to use to populate a memory table  when mysqld starts, for ex: the file may contain LOAD DATA INFILE Command.

Maria Storage Engine is a Crash Safe Version of MYISAM.

Overall MySQL Server Options

 

Overall MySQL Server Options

Option Name Global or

memlock Global
Locks the mysqld daemon into server memory. While this can improve performance if the server runs out of RAM, the mysqld daemon will crash.

sort_buffer_size = buffer_size
Session Determines the amount of system memory allocated for SQL sorts. If this size is exceeded, the server will use hard drive space for sorting data.

thread_cache_size = num_of_threads
Global mysqld creates a cache of unused connection threads rather than destroying threads and creating new ones as need.

thread_concurrency = N Global
Only use on Solaris systems. This should typically be set to twice the number of CPUs. It should be tested carefully before using in production.

tmp_table_size Global
Both this and the max_heap_table_size setting are used to determine the maximum size allowed of an in-memory temporary table before it is converted to a MyISAM table. The smallest value for these two settings is the one utilized.

MOCA (MYSQL Optimal Configuration Architechture)

1)  /opt/Mysql/5.1.33           -------------- Software Installed at this Location
2) db01/mysql/mysql01/data --------  Data Directory
Db01/mysql/mysql01/binlogs    ------------- binary logs directory
Db01/mysql/mysql01/admin       ------------ main administration

 

Db01/mysql/mysql01/Backups   ————  Backups
Db01/mysql/mysql01/Trans        ————  Transaction Data Files
Db01/mysql/mysql01/TransLogs  ———-   Trans Log Files.

Backup Strategies and Backup Procedures.

Logical Backup: Taking Backup Online While Database is Running.

Physical Backup: Taking Backup of Database Files or Disk Partitions: Occupies More Space.

Full Backup: Taking Full Backup of Database.

Incremental Backup: Taking Backups of the Database Since the last Backup.  You may take first full Backup and take Incremental Backups to Save Time. Once a Week full backup might be necessary.

Consistent Backup: Taking Backup at an Exact Moment of Time, A Backup Process may take time to Complete
HOT Backup: Taking Backup while Database is Running.
Cold Backup: Taking Backup While Database is Offline.
Point-in-Time-Restore: Restoring Backup for Certain Time and Date.

REPLICATION – - Configuring Slave Backup Server.

Restore Database and Execute a CHANGE MASTER Command to configure the Slave.

LOAD DATA FROM Master – Master to Slave.
Grant Replication Slave on *.* to ‘Slave’ @ ‘Slave-host’ Identified by ‘Password’ ;

2) Edit My.cnf   Configuration file and add the below line

[mysqld]
Server-Id  =  1
Log – bin = Slave_log

Show Master Status;

Partition PO Values Less than (To_Days (‘2009-01-01)),
Partition PO Values Less than (To_Days (‘2009-02-01)),
Partition PO Values Less than (To_Days (‘2009-03-01)),
Partition PO Values Less than (To_Days (‘2009-04-01)),
Partition PO Values Less than (To_Days (‘2009-05-01)),
Partition PO Values Less than MAXVALUE.

Binlog
Partition
CSV
MEMORY
InnoDB
MyISAM
MRG_MYISAM
ndbcluster – DISABLED – needs to be downloaded and Installed in order to work.

INNODB Row Level Locking

First of all ,

Create the Repository (don’t import any of your working copies or anything.. just the clean repository).

to Create a Repository issue the below command.

svnadmin create –fs-type fsfs /srv/svn/repos

if your Svn Backup is saved in .tgz format i.e., backup.tgz then unzip it first using the command

tar -xvzf filename.gz

for ex: tar -xvzf 030001.tgz will unzip the file in your given location with a dumpfile as saved previously by the command script.

then run the below script to restore the Repository from Backup.

svnadmin load <

ex: #svnadmin load /srv/svn/repos < /home/sree/Desktop/030001/dumpfile.txt

then the dumping process will start and dump all the revisions inside the Selected Folder.

check your repository before updating or committing any of your new files.

here are some of the external bash commands if you need for your directory removal with some content in it.

ex: rm -rf repos (will delete main folder and sub folders inside)

if you have any issues please leave a comment.

to set the Cron Job for SVN use the below commands

#crontab -l will list out all the cron jobs

#crontab -e    will edit the cron job file in VI Editor.

“MAILTO=chava.sree@gmail.com”
0 3 * * * /srv/svn/backup/inc_backup.sh

will do incremental backups everyday.

this is the Reference URL  http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/

Thanks
Sree.

Os: Fedora Linux
Webserver : Apache ver 2.1.2

Issue:
Figured out Apache Server is not running when i run the command “service httpd status” until then i am running http://localhost/ in the browser which is working fine and unable to run any other programs

and when i used this command “# service httpd status: i got the below message

[root@hirodexdbserver ~]# service httpd status
httpd dead but subsys locked

i thought it might be some locks issue and restarted the server

#service httpd restart – i got the below error again.
**********************
[root@hirodexdbserver ~]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
You have mail in /var/spool/mail/root
[root@hirodexdbserver ~]# service httpd restart
**********************

Steps to Resolve the Issue
——————————–
after figuring out and looking at my httpd.conf file and make sure the pid file is pointing to the right file

PidFile run/httpd.pid

and change the path here to
go to /etc/rc.d/init.d/httpd open this file
and find the below line.

pidfile=${PIDFILE-/var/run/httpd/httpd.pid}

to:
pidfile=${PIDFILE-/var/run/httpd.pid}

Restart the Apache Server

# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

which should give you the above status.

hope this helps.

Thanks

#!/bin/bash

echo “+++ Backing up subversion repositories”
NOW=`date +”%OH%M%S”`
cd backup
mkdir -p $NOW
svnadmin dump /srv/svn/repos > $NOW/dumpfile.txt
tar cfpz $NOW.tgz $NOW .
rm -rf $NOW
echo “+++ Svn Backup Successfully Completed. Thank you”

http://bash.cyberciti.biz/backup/copy-all-local-files-to-remote-ftp-server-2/

BACKUP

# Subversion Repositories backup script

# Please start configuring here
svnbasedir=”/srv/svn”
svnfullbkdir=”/srv/svn/backups/full” #Don’t put a trailing slash!
logfile=”/srv/svn/backups/log.txt”
tmpdir=”/tmp”

# dont change below

#check if log file exists
touch ${logfile}
#redirect stdout and stderr to the logfile
mkdir -p $(dirname ${logfile})
exec >> ${logfile}
exec 2>&1

echo “+++ Backing up subversion repositories”
#Looks for dirs containing a file “format”, that’s not in the /db/ directory.
#This seems to match the root of a SVN repo. YMMV
repos=$(find $svnbasedir -name format -a \! -path ‘*/db/*’ -printf ‘%h\n’)

#Make sure that our backup location exists
mkdir -p ${svnfullbkdir}

for repo in $repos ; do
subpath=$(echo ${repo} | sed -e “s#${svnbasedir}/##”)
name=$(basename $subpath)
subpath=$(dirname $subpath)
echo ” $repo”
lastrev=$(svnlook youngest ${repo})

mkdir -p ${svnfullbkdir}/${subpath}
svnadmin dump -q ${repo} > ${svnfullbkdir}/${name}.dump
echo “Dumping ${repo} to ${name}.dump \n”

done

RESTORE

#!/bin/sh
# Subversion Repositories backup script

# Please start configuring here

svnbasedir=”/svn/svn”
svnfullbkdir=”/tmp/svn-dump” #Don’t put a trailing slash!
svnincbkdir=”/subversion/backup/incremental”
logfile=”/subversion/backup/log.txt”
tmpdir=”/tmp”

# dont change below

touch ${logfile}
#redirect stdout and stderr to the logfile
mkdir -p $(dirname ${logfile})
exec >> ${logfile}
exec 2>&1

echo “+++ Restoring subversion repositories “
#Looks for subversion backup dumps taken before
repos=$(find $svnfullbkdir -name ‘*.dump’ )

mkdir -p ${svnbasedir}

for repo in $repos ; do

dirname=${repo%.dump}
name=$(basename $dirname)
$curpath = “/svn/svn/”.${name}
echo ” creating reposity ${name} at ${curpath} “
svnadmin create /svn/svn/${name}

echo ” loading ${repo} “
svnadmin load -q /svn/svn/${name} < ${repo}

done

***** ANOTHER WAY OF BACKUP ***********

http://www.linuxquestions.org/questions/linux-server-73/svn-server-backup-574111/

incremental backup..

# svnadmin dump –incremental /srv/svn/repos | gzip -c > /srv/svn/dumped.gz

Actual Code

#!/bin/bash

NOW=`perl -e ‘print time;’`
cd /var/backup/svn
mkdir $NOW
svnadmin dump /opt/repositories.svn/CC > $NOW/dumpfile.txt
tar cfpz $NOW.tgz $NOW
rm -rf $NOW

…then follow that up with another cronjob that does this:

CRON JOB

45 3 * * * for i in `find /var/backup -type f -mtime +2` ; do rm -f $i ; done

The first cronjob does a full dump of the svn database and then zips it. (We are paranoid, we only do full dumps of databases and database-like applications.) Later on after that’s complete, the nightly backup process runs and the full dump is taken as part of a daily backup to tape. The second cronjob deletes anything it finds in the /var/backup tree that’s older than two days. That way I know I have two days of backups on disk, and I can restore the full database from any day I have tape for.

How to Add a script to Cron Job.
——————————–

Despite the fact DBAs are protectors of an organization’s knowledge, and privy to much confidential information, there is no clearly defined set of rules and standards to help govern and guide their ethical conduct.  Brad McGehee, in an article based on a chapter from the second edition of his book How to Become an Exceptional DBA, discusses a Code of Conduct for DBAs. He advises on why such a code is required, the sort of topics it should contain, and how it might be enforced. Much of this article has relevance for any IT professional.

Physicians, attorneys, accountants, engineers, realtors, and many other professionals have written guidelines designed to discourage misconduct and illegal activity, and to promote the ethical conduct of its members. In fact, according to the Sarbanes-Oxley act, all public companies in the United States are required to create and follow their own code of conduct.

if, as a DBA, you
identify criminal
behavior within
your organization,
what do you do?

However, despite the fact that DBAs are essentially protectors of an organization’s knowledge, and privy to much confidential information, there are no clearly-defined set of rules, values, standards, and guidelines to help govern and guide their behavior.

In this article, I define what a code of conduct is, explain how it can be useful to DBAs and the organizations they work for, and consider if and how it could be enforced. Finally, I offer my take on a “Code of Conduct” for the Exceptional DBA. Rather than be prescriptive, my goal is simply to offer advice to DBAs on how they might conduct themselves within the bounds of the professional responsibilities of their job.

What is a Code of Conduct?

Generally speaking, a code of conduct is a set of formal, written rules, policies, standards, guidelines, obligations, behaviors, expectations, and principles that are voluntarily adhered to by a group of people with similar goals, values, and responsibilities. While other definitions exist, this one will serve our purpose well enough, as we seek to define the standards of conduct to which a DBA should adhere, while carrying out their professional duties.

How Can a Code of Conduct be Useful to DBAs?

To many people, a “Code of Conduct” means one thing: more rules. Most DBAs, myself included, would consider themselves mature enough to make their own decisions and choices, and are naturally resistant to the idea of someone telling them what to do and how to do it.

If this is the case, then why have I written this article? Well, first, my intent is not to offer a set of rules that have to be followed but, instead, a set of guidelines that can be useful for DBAs who want to fully understand and appreciate the nature and scope of their duties and responsibilities. In other words, it is intended to be educational.

It’s a sad fact that
many organizations
don’t fully understand
and appreciate the
true role of a DBA.

At a fundamental level, most DBAs believe they understand what it means to behave “ethically”. In my experience, instances of willful negligence, or blatantly unethical behavior, are relatively scarce in our profession. However, there are many “grey” areas where a DBA, especially a less-experienced DBA, can find themselves unsure of where their responsibilities end, and how exactly they should respond to a potentially compromising situation.

For example if, as a DBA, you identify criminal behavior within your organization, what do you do? Hopefully, your answer would be that you’d report it. However, is that the end of your responsibilities? What if the company you work for does not respond appropriately? What further action should you take, if any?

It’s a sad fact that many organizations don’t fully understand and appreciate the true role of a DBA. In their attempts to save money, they sometimes cut corners that can directly affect the integrity of their data, and so unwittingly place the guardians of their data in a difficult dilemma. If you find yourself in this situation then, hopefully, this code of conduct may help convince your organization of the importance of the role of the DBA, and the need to employ experienced DBAs to safeguard their business data. For example, if you are having difficulty convincing your manager to adhere to government regulation that affects the data you manage, then referring the manager to this Code of Conduct might be useful.

Perhaps above all else, the “Exceptional DBA’s Code of Conduct” can be a source of pride to all of us who choose to adhere to it, and give us more confidence that we are doing the best we can at our jobs.

How Should a Code of Conduct be Implemented and Enforced?

Whenever the topic of a “code of conduct” is discussed, one of the first questions to be asked is: how can it be enforced? After all, if the code is not enforced in any way then, it is argued, who is it benefiting? There is no straightforward answer to the question of how to enforce such a code, or even if it should be enforced, rather than be voluntary.

One suggestion is that the code be enforced by an independent, professional organization, such as the ACM, AITP, or PASS. Anyone who fails to follow the established code of conduct could have their membership of the organization revoked. On the assumption that most businesses would regard membership of the organization a key requirement in their hiring process, then it could, in theory, prove an effective deterrent to malpractice.

Again, any DBA who fails
to follow the established
code could have their
certification rescinded.

Another suggestion is that organizations that offer DBA certifications, such as Microsoft, should enforce some code of conduct. Again, any DBA who fails to follow the established code could have their certification rescinded. Obviously, this would only be an effective measure for organizations that require certification as part of their terms of employment. On the other hand, if Microsoft were to attempt this, what about the Oracle and MySQL DBAs? Would they need a separate code that is tied to their certifications?

Another option is to have a DBA Code of Conduct enforced on a per-organization basis. After all, many organizations already require their employers to abide by a “code of conduct,” often in the form of an employee handbook. Such a code could be expanded to incorporate conduct specific to DBAs. On the other hand, if a company creates a separate code of conduct for DBAs, wouldn’t it also have to create different codes of conducts for other job titles? Again, this is not a great solution.

In light of the numerous complications of implementing and enforcing a DBA’s Code of Conduct, I suggest that following such a code, or not, is a personal choice on behalf of the individual DBA. As I noted earlier, I regard such as code as being primarily educational in nature, with the rewards for following it being a successful, rewarding, long-term, and satisfying career as a DBA. For those who choose not to follow it, well, who knows how their career will turn out? They might get lucky and survive over the long haul, but I doubt if they will ever fall into the Exceptional DBA category.

The Exceptional DBA’s Code of Conduct

This section contains my suggestions for the core items that should comprise “The Exceptional DBA’s Code of Conduct”. My focus is on providing general guidelines that will help any DBA become more successful in their career.

I won’t be so presumptuous as to imply that these guidelines are ideal or exhaustive, but they should hopefully get you thinking and be useful as a guide to you in your role as a DBA.

Protection and Disclosure of Data

1. DBAs shall never intentionally do anything that contributes to the corruption or loss of an organization’s data. While this should be a no-brainer, I am including the obvious, just to cover all the bases.
2. Above all else, the DBA is the guardian, or protector, of an organization’s data. The responsibilities of the DBA in this regard include but are not limited to:
   • Preventing data corruption (from applications, hardware, and people).
   • Provide high data availability, as defined by the organization’s policy. Develop, and regularly test, a disaster recovery plan to minimize unexpected downtime, as defined by the organization’s policy.
3. DBAs shall take all reasonable precautions to ensure that changes to a production server have no detrimental consequences. All proposed changes should be thoroughly tested on a test server prior to deployment to production.
4. DBAs shall proactively take all the necessary steps to ensure that data can only be accessed by authorized users. This may include, but is not limited to: SQL Server security, Windows Authentication security, encryption, SQL injection attack prevention, auditing, or other such security policies, as defined by their organization.
5. DBAs generally have full access to all the data in a database. This presents two important points:
   • DBAs shall only view data in a database if they need to access it to perform their job duties. For example, a DBA may need to access data to optimize a query or create a report. But if the DBA doesn’t need to access and view data for job-related tasks, then DBAs shall not snoop through the data for the sake of viewing it.
   • If, through the course of their required job duties, DBAs do view confidential data, DBAs shall not share any sensitive or confidential information with any unauthorized users.
6. DBAs shall provide full cooperation with both inside and outside audits, providing truthful and factual information
7. If the DBA learns that contractors, co-workers, or managers are making poor choices that could negatively affect the integrity or security of the organization’s data, then the DBA should notify the appropriate persons within their organization.
8. If a DBA discovers misuse of an organization’s data, then the DBA should report this behavior to the appropriate persons within their organization.
9. The DBA should maintain proper and complete documentation for all servers and processes for which he or she is responsible. The documentation should be of a form and standard that will allow a qualified peer to undertake the task, in the DBA’s absence. The DBA should never protect (hide) information from their organization that would prevent another person from easily taking over his or her position, should the need arise.

DBAs and the Law

1. Often, the data under the care of a DBA is subject to laws and regulations of one or more governing bodies. The DBA shall become familiar with these laws and regulations, and adhere to them.
2. The DBA shall enforce all software licensing agreement with vendors, and notify the proper people in the organization if any breech is discovered.
3. If the DBA becomes aware of illegal activity within their organization, the DBA shall notify the proper people in their organization, or the appropriate law enforcement agency.

Dealing With Third-Parties

1. DBAs shall not accept gifts or favors from vendors if the purpose of the gift or favor is to influence, or to reward, the DBA’s choice of purchasing goods and services from the vendor. More specifically:

• • The DBA shall proactively learn their organization’s policies about gifts, so they know what they can or cannot accept. For example, company policy might allow a vendor to take a DBA to dinner, or to receive a free gift at a conference or a user’s group meeting. On the other hand, an organization’s policy might limit the amount of the gift to a maximum value, such as $50.00.
  • The acceptance of a gift, even if allowed by the organization, should not influence the behavior of the DBA in regards to potential purchase of the vendor’s products and services.
  • Whenever a DBA purchases, or influences the purchase of goods or services, the DBA should only purchase those good or services because they are the ones that best meet the needs of the organization.

General Conduct

1. The DBA shall conduct himself, or herself, professionally in all relationships with the people they deal with on a daily basis, including managers, co-workers, vendors, and their “internal customers.” Favoritism and discrimination of any sort is to be avoided.
2. If a DBA makes a mistake, and all DBAs make a mistake at one time or another, the mistake should be fixed as soon as is practical, and the DBA should inform everyone who might be negatively affected by the mistake as soon as possible. In addition:
   • Mistakes should never be covered up.
   • DBAs should take full responsibility for all their decisions.
3. If a DBA has management responsibilities, the DBA should mentor those who need help accomplishing their objectives, and allow those with experience and enthusiasm to succeed and flourish within the environment he or she provides for them.
4. The DBA will do their job to the best of his or her ability, developing and implementing best practices as appropriate, to help ensure the smooth and successful operations of the organization’s databases.
5. IT technology changes quickly. It is the responsibility of the DBA to proactively keep up on all technical areas that directly affect his or her job.
6. If the DBA has agreed to a SLA (Service Level Agreement) for the servers under their care, then they will abide by those agreements. If they are not technically able to abide by the agreed upon SLA (for whatever reason), the DBA needs to contact the SLA owner and work out how to resolve the problem.
7. Occasionally, the DBA may find that there is no established policy with regard to a particular aspect of their duties. For example, perhaps an organization doesn’t have a policy that defines the desired level of uptime, or that defines who can access what data. In areas where policies should be established (to ensure the protection and integrity of the organization’s data), but are not established, the DBA shall proactively identify the areas of need, make recommendations, and present them to the organization’s management so they can make appropriate decisions.

It is impossible to provide advice to guide a DBA through every aspect of their duties and responsibilities as a DBA, and there are many “grey areas”. What does the DBA do when they notify the appropriate persons of an important issue that could negatively impact the organization’s data, and no action is taken to rectify the issue? Where do a DBA’s responsibilities end with regard to uncovering illegal activity within an organization? What does a DBA do when a cost-cutting measure means they feel they can’t fully adhere to their own code of conduct? These are all difficult question for any employee, not just a DBA.

If you, as a DBA, feel “pressured” to do something you are uncomfortable doing, then the best advice I can give is to talk to people. In most cases, the first step is to tell your manager, co-worker, project manager, or whoever is causing the problem, that you feel uncomfortable with their request, action or inaction, and explain why. If they don’t want to listen, your next option is to escalate the issue to their manager. You may, or may not, receive management support for your case. If not, then you have to decide to “give in,” or to find another job.

The key thing is that the DBA needs to keep cool and not take any rash action. Before making an important decision that could potentially hurt your career, talk over the issue with people you trust in order to get their feedback and input. Only take action once you have clearly and thoughtfully considered the consequences of your choices. This might mean keeping quiet and not making trouble, escalating the issue to higher levels of management, reporting a crime to an enforcement agency, or finding a new job.

Summary

My goal in suggesting an “Exceptional DBA’s Code of Conduct” is to provide advice and guidance for those DBAs wanting to learn more about what it takes to become an Exceptional DBA. Is the code complete or perfect? Of course not. Consider it a working, educational document; it is certainly not intended to be an absolute set of rules that every DBA must always follow.

My hope is that it serves as a useful guide to the DBAs looking to navigate the “grey areas” and as a foundation for all those entering the profession with ambitions to become Exceptional DBAs.

When I am trying to Create a Function in MySql Query Browser, I had the below error and want to get rid of it.. and found a solution, as

  ERROR 1418 (HY000) at line 253: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration
               and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)

for the above error just set the below parameters in my.ini (for windows) and my.cnf (in linux) files to edit and add the below text.

in windows to edit the my.ini file go here c:/program files/mysql/mysql version5.4/bin/my.ini

in linux to edit the my.cnf file go here /etc/mysql/bin/my.cnf and add the below line to your config file.

 log_bin_trust_function_creators=1

http://www.eukhost.com/forums/f15/how-install-subversion-svn-linux-server-vps-3737/

http://marcgrabanski.com/article/installing-subversion-on-apache


yum install subversion


yum install mod_dav_svn

Creating a User in Svn using the bash Command

1) htpasswd -cm users sam
2) htpasswd -cm users chava
3) htpasswd -cm users veera

Importing a Folder to Repository in Svn (to checkout for later use)

1) svn import -m "Initial Import" local-directory-projectX http://server-name/svn/projectX

2)

follow this link to install subversion Server 1.4 on Linux Server with Apache WebServer Running

http://www.yolinux.com/TUTORIALS/LinuxSubversionAndTracServer.html

Create a Subversion repository using the FSFS database format: (as root)

• mkdir /srv/svn/repos
• svnadmin create --fs-type fsfs /srv/svn/repos
• chown apache:apache -R /srv/svn/repos
  (The Apache web server running as user “apache” is the process owner which will be interfacing with the Subversion repository.)
• SELinux security attribute assignment: (Red Hat EL4+, Fedora 3+)
  restorecon -R /srv/svn/repos

Apache configuration file: /etc/httpd/conf.d/subversion.conf

• This example performs no authentication:
  

  LoadModule dav_svn_module modules/mod_dav_svn.so - Required for Apache interface LoadModule authz_svn_module modules/mod_authz_svn.so - Required for "per-directory" access control <Location /svn> - URL path in web browser after "http://domain.com/" DAV svn SVNPath /srv/svn/repos - Local path on server filesystem </Location>

• This example authenticates to a local Apache user password file: User logins and passwords have no connection to user accounts.
  

  LoadModule dav_svn_module modules/mod_dav_svn.so LoadModule authz_svn_module modules/mod_authz_svn.so <Location /svn> DAV svn SVNPath /srv/svn/repos AuthType Basic AuthName "Subversion Repository" AuthUserFile /etc/httpd/conf/userpw require valid-user </Location>

  Remove the line “require valid-user” to allow read-only access for unauthenticated users.

Importing Directory of Files Into Subversion:

Subversion Repository on Server File system:

• /srv/svn/repos

Import directory of files into Subversion:

• Local directory to import into Subversion:
  • projectX/trunk/...source code goes here...
  • projectX/branches
  • projectX/tags
• svn import -m "Initial Import" local-directory-projectX http://server-name/svn/projectX
  The import will create the new subdirectory branch “projectX” in the repository.

Alternate directory creation method: (repository exists but the directories need to be generated)

• svn mkdir -m"Initial creation of trunk directory" http://svn-server/svn/projectX/trunk
• svn mkdir -m"Initial creation of branches directory" http://svn-server/svn/projectX/branches
• svn mkdir -m"Initial creation of tags directory" http://svn-server/svn/projectX/tags

Browser view of a typical repository directory schema:

• http://svn-server/svn/projectX/trunk: For the “HEAD”
• http://svn-server/svn/projectX/branches: For branches or “forks” in the code.
• http://svn-server/svn/projectX/tags: For tags identifying snapshots of milestones or significant releases.

• /trunk : Contains the current development branch. New projects should be started here.
• /branches : A branch is just a special copy of the source tree within Subversion that started with a specific revision of the code. A branch is created by using “svn copy” of the trunk to a branch i.e. /branch/name_of_branch_1.0.
• /branches/personal : Personal branches are for doing some work which you don’t want to have interfere with the main trunk until it is working or better defined.
• /tags : This is a ethod of bookmarking or taking a “snapshot” of some work in progress.
• /releases : This is similar to “tags” except that this version of the code makes it to production. Names used here should match product release names. i.e. version_1.1

Avoiding false diffs due to “^M” in a cross platform environment:

The Microsoft development tools love to add a “^M” at the end of the line of every file they edit. This breaks UNIX shell script and causes many file difference tools to show a difference in a line of code. This can be avoided in a cross platform environment by telling Subversion that certain files (or all files) that no “^M”’s should be appended at the end of a line. The trigger in fact removes the “^M” when the file is checked in. It can also be removed using the command dos2unix.

Import files to support no “^M” carriage returns at the end of lines:
(This sets file properties and creates a check-in trigger.)

• Remove “^M” from files: find local-directory -name "*.cpp" -exec dos2unix {} \;
• Upload directory of files into Subversion:
  svn import -m "Initial Import" local-directory http://server-name/svn/projectX
• Checkout files from repository: svn co http://server-name/svn/projectX/trunk
  This creates your local working directory under Subversion control.
• Set file properties such that “^M” are removed during check-in if added:
  find ./ -name "*.cpp" -exec svn propset svn:eol-style LF {} \;
• Apply property changes to repository: svn ci -m "Apply LF properties" local-directory

Note that the “propset” command must be used for new files added to the Subversion repository if they are to have these properties.

Users can also set this option in the file: $HOME/.subversion/config

..
...

[auto-props]
README = svn:eol-style=native
INSTALL = svn:eol-style=native
*.c = svn:eol-style=LF
*.cpp = svn:eol-style=LF
*.h = svn:eol-style=LF
*.dsp = svn:eol-style=CRLF
*.dsw = svn:eol-style=CRLF
*.sh = svn:eol-style=native;svn:executable
*.txt = svn:eol-style=LF
*.png = svn:mime-type=image/png
*.jpg = svn:mime-type=image/jpeg
Makefile = svn:eol-style=LF
*.html = svn:eol-style=LF
*.css = svn:eol-style=LF
*.java = svn:eol-style=LF
*.xml = svn:eol-style=LF
*.m4 = svn:eol-style=LF
*.pdf = svn:mime-type=application/pdf

...
..
Data repository dump:

Dump first entry to current:
 svnadmin dump /srv/svn/repos --revision 0:HEAD > repos.dump
Dump revision 625 to current:
 svnadmin dump /srv/svn/repos --revision 625:HEAD --incremental > repos-625.dump

*************************************************************************************************************


TROUBLESHOOTING SUBVERSION.
---------------------------

use the below path 

run the below commands first to make sure you are giving access to the repos folder.

use these vi commands to edit the file.

vi filename.

examples: http://www.cs.colostate.edu/helpdocs/vi.html

chown apache:apache -R /srv/svn/repos

restorecon -R /srv/svn/repos

edit this file and add the below script.

/etc/httpd/conf.d/svnserve.conf

enable these statements below

anon-access = read
auth-access = write

<Location /svn>
 DAV svn
 SVNPath /srv/svn/repos
 AuthType Basic
 AuthName "Subversion Repository"
 AuthUserFile /srv/svn/access/users
 require valid-user
</Location>

Search for something in Linux

rpm -q -a | grep mysql   (searches for mysql files)
rpm -q -l mysql          (lists out the path of mysql search string) 

Load Putty first and Connect to the Remote Host using Remote Username and Password

1) pwd : to locate the absolute path
2) Ls -l   : list all the directory info.
3) Ls : list all the directory info in wide format.
4)  ./  : used to execute shell script files with “.sh” extension.
5) cp : copy directory from one path to another
6) cd .. : change directory just like any other Dos Directory.

htpasswd -m /etc/.svn-auth-file <username> — to create user account(must be root)  is to create a username and password from apache to use with Subversion.

http://marcgrabanski.com/article/installing-subversion-on-apache (subversion installation instructions).

actual path is    ”htpasswd -m /srv/svn/access/users  username  (pressing enter will ask for password followed by password confirmation and will be saved into the access file.

# service httpd restart
(will restart the apache server).

A Comprehensive Database Security Model

This week I am taking a bit of a departure. Normally I write about things I have already done, but this week I want to speculate a bit on a security model I am thinking of coding up. Basically I have been asking myself how to create a security model for database apps that never requires elevated privileges for code, but still allows for hosts sharing multiple applications, full table security including row level and column level security, and structural immunity to SQL injection.

The Functional Requirements

Let’s consider a developer who will be hosting multiple database applications on a server, sometimes instances of the same application for different customers. The applications themselves will have different needs, but they all boil down to this:

• Some applications will allow surfers to join the site and create accounts for themselves, while others will be private sites where an administrator must make user accounts.
• Some applications will not contain sensitive data, and so the site owner wants to send forgotten passwords in email — which means the passwords must be stored in plaintext. Other site owners will need heightened security that disallows storing of passwords in plaintext.
• In both cases, administrators must of course be able to manage accounts themselves.
• The system should be structurally immune to SQL injection.
• It must be possible to have users with the same user id (“Sheilia”, “John”, etc.) on multiple applications who are actually totally different people.
• The application code must never need to run at an elevated privelege level for any reason — not even to create accounts on public sites where users can join up and conduct transactions.
• It must be possible for the site owners or their agents to directly connect to the database at very least for querying and possibly to do database writes without going through our application.
• Users with accounts on one app must never be able to sign on to another app on the same server.

These requirements represent the most flexible possible combination of demands that I have so far seen in real life. The question is, can they be met while still providing security? The model I’d like to speculate on today says yes.

Informed Paranoia Versus Frightened Ignorance

Even the most naive programmer knows that the internet is not a safe place, but all too often a lot of security advice you find is based on frightened ignorance and takes the form, “never do x, you don’t know what might happen.” If we are to create a strong security model, we have to do better than this.

Much better is to strive to be like a strong system architect, whose approach is based on informed paranoia. This hypothetical architect knows everybody is out to compromise his system, but he seeks a thorough knowledge of the inner workings of his tools so that he can engineer the vulnerabilities out as much as possible. He is not looking to write rules for the programmer that say “never do this”, he is rather looking to make it impossible for the user or programmer to compromise the system.

Two Examples

Let us consider a server hosting two applications, which are called “social” and “finance”.

The “social” application is a social networking site with minimal security needs. Most important is that the site owners want members of the general public to sign up, and they want to be able to email forgotten passwords (and we can’t talk them out of it) — so we have to store passwords in plaintext.

The “finance” application is a private site used by employees of a corporation around the world. The general public is absolutely not welcome. To make matters worse however, the corporation’s IT department demands to be able to directly connect to the database and write to the database without going through the web app. This means the server will have an open port to the database. Sure it will be protected with SSL and passwords, but we must make sure that only users of “finance” can connect, and only to their own application.

Dispensing With Single Sign-On

There are two ways to handle connections to a database. One model is to give users real database accounts, the other is to use a single account to sign on to the database. Prior to the web coming along, there were proponents of both models in the client/server world, but amongst web developers the single sign-on method is so prevalent that I often wonder if they know there is any other way to do it.

Nevertheless, we must dispense with the single sign-on method at the start, regardless of how many people think that Moses carved it on the third tablet, because it just has too many problems:

• Single Sign-on is the primary architectural flaw that makes SQL injection possible. As we will see later, using real database accounts makes your site (almost) completely immune to SQL injection.
• Single Sign-on requires a connection at the maximum privilege level that any system user might have, where the code then decides what it will let a particular user do. This is a complete violation of the requirement that code always run at the lowest possible privilege level.
• Single Sign-on totally prevents the requirement that authorized agents be allowed to connect to the database and directly read and write values.

So single sign-on just won’t work with the requirements listed. This leads us to creating real accounts on the database server.

Real Accounts and Basic Security

When you use a real database account, your code connects to the database using the username and password provided by the user. Anything he is allowed to do your code will be allowed to do, and anything he is not allowed to do will throw and error if your code tries to do it.

This approach meets quite a few of our requirements nicely. A site owner’s IT department can connect with the same accounts they use on the web interface — they have the same privileges in both cases. Also, there is no need to ever have application code elevate its privilege level during normal operations, since no regular users should ever be doing that. This still leaves the issue of how to create accounts, but we will see that below.

A programmer who thinks of security in terms of what code can run will have a very hard time wrapping his head around using real database accounts for public users. The trick to understanding this approach is to forget about code for a minute and to think about tables. The basic fact of database application security is that all security resolves to table permissions. In other words, our security model is all about who can read or write to what tables, it is not about who can run which program.

If we grant public users real database accounts, and they connect with those accounts, the security must be handled within the database itself, and it comes down to:

• Defining “groups” as collections of users who share permissions at the table level.
• Deciding which groups are allowed select, insert, update, and delete privileges on which tables.
• Granting and revoking those privileges on the server itself when the database is built.
• At very least row-level security will be required, wherein a user can only see and manipulate certain rows in a table. This is how you keep users from using SQL Injection to mess with each other’s order history or member profiles.
• Column security is also very nice to finish off the picture, but we will not be talking about that today as it does not play into the requirements.

Now we can spend a moment and see why this approach eliminates most SQL Injection vulnerabilities. We will imagine a table of important information called SUPERSECRETS. If somebody could slip in a SQL injection exploit and wipe out this table we’d all go to jail, so we absolutely cannot allow this. Naturally, most users would have no privileges on this table — even though they are directly connected to the database they cannot even see the table exists, let alone delete from it. So if our hypothetical black hat somehow slips in “;delete from supersecrets” and our code fails to trap for it, nothing happens. They have no privlege on that table. On the other side of things, consider the user who is privileged to delete from that table. If this user slips in a “;delete from supersecrets” he is only going to the trouble with SQL Injection to do something he is perfectly welcome to do anyway through the user interface. So much for SQL injection.

To repeat a point made above: row-level security is a must. If you grant members of a social site global UPDATE privileges on the PROFILES table, and you fail to prevent a SQL Injection, all hell could break loose. Much better is the ability to limit the user to seeing only his own row in the PROFILE table, so that once again you have created a structural immunity to SQL injection.

Anonymous Access

Many public sites allow users to see all kinds of information when they are not logged on. The most obvious example would be an eCommerce site that needs read access to the ITEMS table, among others. Some type of anonymous access must be allowed by our hypothetical framework.

For our two examples, the “social” site might allow limited viewing of member profiles, while the “finance” application must show absolutely nothing to the general public.

If we want a general solution that fits both cases, we opt for a deny-by-default model and allow each application to optionally have an anonymous account.

First we consider deny-by-default. This means simply that our databases are always built so that no group has any permissions on any tables. The programmer of the “social” site now has to grant certain permissions to the anonymous account, while the programmer of the “finance” application does nothing – he already has a secure system.

But still the “finance” site is not quite so simple. An anonymous user account with no privileges can still log in, and that should make any informed paranoid architect nervous. We should extend the deny-by-default philosophy so the framework will not create an anonymous account unless requested. This way the programmer of the “finance” application still basically does nothing, while the programmer of the “social” must flip a flag to create the anonymous account.

Virtualizing Users

If we are having real database accounts, there is one small detail that has to be addressed. If the “social” site has a user “johnsmith” and the finance application has a user of the same name, but they are totally different people, we have to let both accounts exist but be totally separate.

The answer here is to alias the accounts. The database server would actually have accounts “finance_johnsmith” and “social_johnsmith”. Our login process would simply take the username provided and append the code in front of it when authenticating on the server. ‘nuf said on that.

Allowing Public Users To Join

The “social” site allows anybody to join up and create an account. This means that somehow the web application must be able to create accounts on the database server. Yet it must do this without allowing the web code to elevate its privileges, and while preventing the disaster that would ensue if a user on the “social” site somehow got himself an account on the “finance” site.

Believe it or not, this is the easy part! Here is how it works for the “social” site:

• Create a table of users. The primary key is the user_id which prevents duplication.
• For the social site, there is a column called PASSWORD that stores the password in plaintext.
• Allow the anonymous account to INSERT into this table! (Remember though that deny-by-default means that so far this account has no other privileges).
• Put an INSERT trigger on the table that automatically creates an aliased user account, so that “johnsmith” becomes “social_johnsmith”. The trigger also sets the password.
• A DELETE trigger on the table would delete users if the row is deleted.
• An UPDATE trigger on the table would update the password if the user UPDATES the table.
• Row level security is an absolute must. Users must be able to SELECT and UPDATE table, but only their own row. If your database server or framework cannot support row-level security, it’s all out the window.

This gives us a system that almost gets us where we need to be: the general public can create acounts, the web application does not need to elevate its privileges, users can set and change their passwords, and no user can see or set anything for any other user. However, this leaves the issue of password recovery.

In order to recover passwords and email them to members of the “social” site, it is tempting to think that the anonymous account must be able to somehow read the users table, but that is no good because then we have a structural flaw where a successful SQL injection would expose user accounts. However, this also turns out to be easy. There are two options:

• Write a stored procedure that the anonymous user is free to execute, which does not return a password but actually emails it directly from within the database server. This requires your database server be able to send emails. (Postgres can, and I assume SQL Server can, and I don’t really know about mySql).
• Create a table for password requests, allow inserts to it but nothing else. A trigger sends the email. In this approach you can track email recovery requests.

For the “finance” application we cannot allow any of this to happen, so again we go to the deny-by-default idea. All of the behaviors above will not happen unless the programmer sets a flag to turn them on when the database is built.

This does leave the detail of how users of the “finance” application will reset their passwords. For details on how a secure app can still allow password resets, see my posting of Sept 7 2008 Secure Password Resets.

One More Detail on Public Users

We still have one more detail to handle for public users. Presumably a user, having joined up, has more privileges than the anonymous account. So the web application must be able to join them into a group without elevating its privileges. The solution here is the same as for creating the account: there will be a table that the anonymous user can make inserts into (but nothing else), and a trigger will join the user to whatever group is named.

Except for one more detail. We cannot let the user join whatever group they want, only the special group for members. This requirement can be met by defining the idea of a “freejoin” group and also a “solo” group. If the anonymous user inserts into a user-group table, and the requested group is flagged as allowing anybody to join, the trigger will allow it, but for any other group the trigger will reject the insert. The “solo” idea is similar, it means that if a user is in the “members” group, and that group is a “solo” group, they may not join any other groups. This further jails in members of the general public.

Almost Done: User Administration

In the last two sections we saw the idea of a table of users and a cross-reference of users to groups. This turns out to solve another issue we will have: letting administrators manage groups. If we define a group called “user_administrators” and give them total power on these tables, and also give them CRUD screens for them, then we have a user administrator system. This works for both the “social” and the “finance” application.

The triggers on the table have to be slightly different for the two cases, but that is a small exercise to code them up accordingly.

Cross-Database Access

Believe it or not, the system outlined above has met all of our requirements except one. So far we have a system that never requires the web server to have any elevated priveleges within the database, allows members of the public to join some sites while barring them from others, is structurally immune from SQL injection, allows different people on different sites to have the same user id, and allows administrators of both sites to directly manage accounts. Moreover, we can handle both plaintext passwords and more serious reset-only situations.

This leaves only one very thorny issue: cross-database access. The specific database server I use most is PostgreSQL, and this server has a problem (for this scenario) anyway, which is that out-of-the-box, a database account can connect to any database. This does not mean the account has any priveleges on the database, but we very seriously do not want this to happen at all. If a member of the “social” site can connect to the “finance” app, we have a potential vulnerability even if he has zero privileges in that database. We would be much happier if he could not connect at all.

In Postgres there is a solution to this, but I’ve grown to not like it. In Postgres you can specify that a user can only connect to a database if they are in a group that has the same name as the database. This is easy to set up, but it requires changing the default configuration of Postgres. However, for the sheer challenge of it I’d like to work out how to do it without requiring that change. So far I’m still puzzling this out. I’d also like to know that the approach would work at very least on MS SQL Server and mySql.

Conclusion

Most of what is in this week’s essay is not that radical to any informed database veteran. But to web programmers who were unfortunate enough to grow up in the world of relational-databases-must-die nonsense, it is probably hard or impossible to imagine a system where users are connecting with real database accounts. The ironic thing is that the approached described here is far more secure than any single sign-on system, but it requires the programmer to shift thinking away from action-based code-centric models to what is really going on: table-based privileges. Once that hurdle is past, the rest of it comes easy.

8.3.10. The Privileges Tab

http://dev.mysql.com/doc/workbench/en/wb-table-editor-privileges.html

Security – Roles and Password Expiry on MySQL using SECURICH

http://mysqlpreacher.com/wordpress/2009/06/security-roles-and-password-expiry-on-mysql-using-securich/

Lately there has been quite some talk about security on MySQL, and I’ve decided to GPL a package I wrote, implmenting Roles on MySQL.
This technology has been available on other databases for quite some time, but hasn’t quite yet made it to MySQL’s feature list and apart from this tool, the only solution I know of is google’s patches for MySQL 5.0.

The name is SECURICH and downloads as well as documentation are available athttp://www.securich.com/. The bug list is available at http://www.securich.com/mantis/ (you’ll have to register and log in first). Privileges can be granted on a database or table level with a few combinations like:
1. database as one,
2. all tables in database (useful when you need to grant a bunch of privileges to all tables but a few thus revoking the few later),
3. single tables,
4. stored procedure or
5. tables through the use of regular expression.

The user has the facility to create a role and update it on the fly thus propagating the changes onto already existing users having the role in question. There are also loads of other functionalities available and I urge you to check the documentation onhttp://www.securich.com/.

A cool feature I like is password history which enables dbas to enforce changing of passwords after a certain period of time.

#################################################################################################################################################
## PLEASE NOTE THAT:
## 1. THIS IS AN ALPHA VERSION,
## 2. YOU SHOULD NOT INSTALL THIS PACKAGE ON A PRE-EXISTING MYSQL WHICH ALREADY HAS PRIVILEGES SET UP.
#################################################################################################################################################
##
## This tool will modify current privileges on the “mysql” database tables and should only be used with freshly installed mysqls.
##
##################################################################################################################################################

Features like block user and unblock user as well as many other functionalities are being developed and more features will be added as time goes by.

I would greatly appreciate your feedback about what you think of the package, features you might wish to include etc (feature requests should be done throughhttp://www.securich.com/mantis/ as well.)

]]> http://grabameat.wordpress.com/2009/07/16/roles-and-previleges-in-mysql/feed/ 0 grabameat http://grabameat.wordpress.com/2009/07/15/mysql-what-i-know-as-of-today/ http://grabameat.wordpress.com/2009/07/15/mysql-what-i-know-as-of-today/#comments Wed, 15 Jul 2009 19:33:05 +0000 grabameat http://grabameat.wordpress.com/?p=51 ]]>

1) Downloaded and installed mySql on my Local Machine.

Host: Localhost
Username: ODBC
Password:  nothing

2) Installed MySqlQuery Browser, MySqlAdministrator from Mysql Website

3) Connected to MySql Server Using the Below Parameters

i) use the Command prompt and goto  c:/Program files/mysql/mysqlserver 5.4/bin/mysql.exe
ii) To Connect to a Database   “Use Databasename”  ex: Use Text;
iii) To see a list of tables in mySql just type  ”SHOW TABLES;”
iv) To See Records in a table type “Select * from test;” will display all records.
v)for help with the mysql  type “help” and press enter at the mysql prompt

General SQL Topic

RMAN

Advice on data recovery, parallel backup of the same file, and virtual catalogs for security are just a few of the new gems available from RMAN in Oracle Database 11g.

Download Oracle Database 11g

Data Recovery Advisor

Consider the error shown below:

SQL> conn scott/tiger
Connected.
SQL> create table t (col1 number);
create table t (col1 number)
*
ERROR at line 1:
ORA-01116: error in opening database file 4
ORA-01110: data file 4: '/home/oracle/oradata/PRODB3/users01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3

Does it look familiar? Regardless of your experience as a DBA, you probably have seen this message more than once. This error occurs because the datafile in question is not available—it could be corrupt or perhaps someone removed the file while the database was running. In any case, you need to take some proactive action before the problem has a more widespread impact.

In Oracle Database 11g, the new Data Recovery Advisor makes this operation much easier. The advisor comes in two flavors: command line mode and as a screen in Oracle Enterprise Manager Database Control. Each flavor has its advantages for a given specific situation. For instance, the former option comes in handy when you want to automate the identification of such files via shell scripting and schedule recovery through a utility such as cron or at. The latter route is helpful for novice DBAs who might want the assurance of a GUI that guides them through the process. I’ll describe both here.

Command Line Option

The command line option is executed through RMAN. First, start the RMAN process and connect to the target.

$ rman target=/

Recovery Manager: Release 11.1.0.5.0 - Beta on Sun Jul 15 19:43:45 2007

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: PRODB3 (DBID=3132722606)

Assuming that some error has occurred, you want to find out what happened. The list failure command tells you that in a jiffy.

RMAN> list failure;

If there is no error, this command will come back with the message:

no failures found that match specification

If there is an error, a more explanatory message will follow:

using target database control file instead of recovery catalog
List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
142        HIGH     OPEN      15-JUL-07     One or more non-system datafiles are missing

This message shows that some datafiles are missing. As the datafiles belong to a tablespace other than SYSTEM, the database stays up with that tablespace being offline. This error is fairly critical, so the priority is set to HIGH. Each failure gets a Failure ID, which makes it easier to identify and address individual failures. For instance you can issue the following command to get the details of Failure 142.

RMAN> list failure 142 detail;

This command will show you the exact cause of the error.

Now comes the fun part: How do you rectify the error? Seasoned DBAs will probably ace this without further help but novice DBAs (and even experienced but tired ones) will welcome some guidance here. They can turn to Data Recovery Advisor for assistance:

RMAN> advise failure;

It responds with a detailed explanation of the error and how to correct it:

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
142        HIGH     OPEN      15-JUL-07     One or more non-system datafiles are missing

analyzing automatic repair options; this may take some time
using channel ORA_DISK_1
analyzing automatic repair options complete

Mandatory Manual Actions
========================
no manual actions available

Optional Manual Actions
=======================
1. If file /home/oracle/oradata/PRODB3/users01.dbf was unintentionally renamed or moved, restore it

Automated Repair Options
========================
Option Repair Description
------ ------------------
1      Restore and recover datafile 4
  Strategy: The repair includes complete media recovery with no data loss
  Repair script: /home/oracle/app/diag/rdbms/prodb3/PRODB3/hm/reco_3162589478.hm

This output has several important parts. First, the advisor analyzes the error. In this case, it’s pretty obvious: the datafile is missing. Next, it suggests a strategy. In this case, this is fairly simple as well: restore and recover the file. (Please note that I have deliberately chosen a simple example to focus the attention on the usage of the tool, not to discuss the many cases where the database could fail and how they can be recovered. The dynamic performance view V$IR_MANUAL_CHECKLIST also shows this information.)

However, the most useful task Data Recovery Advisor does is shown in the very last line: it generates a script that can be used to repair the datafile or resolve the issue. The script does all the work; you don’t have to write a single line of code.

Sometimes the advisor doesn’t have all the information it needs. For instance, in this case, it does not know if someone moved the file to a different location or renamed it. In that case, it advises to move the file back to the original location and name (under Optional Manual Actions).

OK, so the script is prepared for you. Are you ready to execute it? I don’t know about you, but I would verify what the script actually does first. So, I issue the following command to “preview” the actions the repair task will execute:

RMAN> repair failure preview;

Strategy: The repair includes complete media recovery with no data loss
Repair script: /home/oracle/app/diag/rdbms/prodb3/PRODB3/hm/reco_741461097.hm

contents of repair script:
   # restore and recover datafile
   sql 'alter database datafile 4 offline';
   restore datafile 4;
   recover datafile 4;
   sql 'alter database datafile 4 online';

This is good; the repair seems to be doing the same thing I would have done myself using RMAN. Now I can execute the actual repair by issuing:

RMAN> repair failure;

Strategy: The repair includes complete media recovery with no data loss
Repair script: /home/oracle/app/diag/rdbms/prodb3/PRODB3/hm/reco_3162589478.hm

contents of repair script:
   # restore and recover datafile
   sql 'alter database datafile 4 offline';
   restore datafile 4;
   recover datafile 4;
   sql 'alter database datafile 4 online';

Do you really want to execute the above repair (enter YES or NO)?

Assuming I’m OK, I answer YES and the action goes on:

executing repair script

sql statement: alter database datafile 4 offline

Starting restore at 15-JUL-07
using channel ORA_DISK_1

channel ORA_DISK_1: restoring datafile 00004
input datafile copy RECID=5 STAMP=628025835 file name=/home/oracle/flasharea
    /PRODB3/datafile/o1_mf_users_39ocxbv3_.dbf
destination for restore of datafile 00004: /home/oracle/oradata/PRODB3/users01.dbf
channel ORA_DISK_1: copied datafile copy of datafile 00004
output file name=/home/oracle/oradata/PRODB3/users01.dbf RECID=0 STAMP=0
Finished restore at 15-JUL-07

Starting recover at 15-JUL-07
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 51 is already on disk as file /home/oracle/
   flasharea/PRODB3/archivelog/2007_07_15/o1_mf_1_51_39ocxxdw_.arc
... and so on ...
name=/home/oracle/flasharea/PRODB3/archivelog/2007_07_15/o1_mf_1_55_39ocy9ox_.arc thread=1 sequence=55
media recovery complete, elapsed time: 00:00:01
Finished recover at 15-JUL-07

sql statement: alter database datafile 4 online
repair failure complete

RMAN>

Note how RMAN prompts you before attempting to repair. In a scripting case, you may not want to do that; rather, you would want to just go ahead and repair it without an additional prompt. In such a case, just use repair failure noprompt at the RMAN prompt.

Proactive Health Checks

It helps you sleep better at night knowing that the database is healthy and has no bad blocks. But how can you ensure that? Bad blocks show themselves only when they are accessed so you want to identify them early and hopefully repair them using simple commands before the users get an error.

The tool dbverify can do the job but it might be a little inconvenient to use because it requires writing a script file contaning all datafiles and a lot of parameters. The output also needs scanning and interpretation. In Oracle Database 11g, a new command in RMAN, VALIDATE DATABASE, makes this operation trivial by checking database blocks for physical corruption. If corruption is detected, it logs into the Automatic Diagnostic Repository. RMAN then produces an output that is partially shown below:

RMAN> validate database;

Starting validate at 09-SEP-07
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=110 device type=DISK
channel ORA_DISK_1: starting validation of datafile
channel ORA_DISK_1: specifying datafile(s) for validation
input datafile file number=00002 name=/home/oracle/oradata/ODEL11/sysaux01.dbf
input datafile file number=00001 name=/home/oracle/oradata/ODEL11/system01.dbf
input datafile file number=00003 name=/home/oracle/oradata/ODEL11/undotbs01.dbf
input datafile file number=00004 name=/home/oracle/oradata/ODEL11/users01.dbf
channel ORA_DISK_1: validation complete, elapsed time: 00:02:18
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
1    OK     0              12852        94720           5420717
  File Name: /home/oracle/oradata/ODEL11/system01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              65435
  Index      0              11898
  Other      0              4535            

File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
2    OK     0              30753        115848          5420730
  File Name: /home/oracle/oradata/ODEL11/sysaux01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              28042
  Index      0              26924
  Other      0              30129           

File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
3    OK     0              5368         25600           5420730
  File Name: /home/oracle/oradata/ODEL11/undotbs01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              0
  Index      0              0
  Other      0              20232           

File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
4    OK     0              2569         12256           4910970   

... <snipped> ...

Otherwise, in case of a failure you will see on parts of the above output:

List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
7    FAILED 0              0            128             5556154
  File Name: /home/oracle/oradata/ODEL11/test01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              108
  Index      0              0
  Other      10             20

You can also validate a specific tablespace:

RMAN> validate tablespace users;

Or, datafile:

RMAN> validate datafile 1;

Or, even a block in a datafile:

RMAN> validate datafile 4 block 56;

The VALIDATE command extends much beyond datafiles however. You can validate spfile, controlfilecopy, recovery files, Flash Recovery Area, and so on.

Enterprise Manager Interface

Let’s see how failures can also be detected and corrected via Enterprise Manager.

First go to the Database homepage; top part is shown below.

Scroll down further until you see the Alerts section, as shown below:

This window shows the actual RMAN command to be issued. At this time you can press Submit to execute the RMAN job. Note the contents of the window: It’s a real RMAN command, which you can copy and paste into an RMAN prompt.

The Enterprise Manager interface to RMAN provides the best of both worlds: the power of RMAN without the complexities of its command language. Advanced users of RMAN may not find it that useful, but for a novice, it’s a lifesaver—especially when considering how a relatively complicated block media recovery was done using a simple interface.

Flashback Logs to Rescue

Remember Flashback Logging introduced in Oracle Database 10g? It records the optimized versions of the before-images of changed blocks into flashback logs that are generated in the Flashback Recovery Area, provided Flashback has been enabled in the database. These logs help you to flash the database back to a point in the time in the past without doing a point-in-time recovery from backups.

Well, since these flashback logs contain past images of the blocks, why not use them for recovery as well? Oracle Database 11g does exactly that. When you recover the specific block (or blocks), Oracle looks in the Flashback logs (instead of datafiles) to find a good copy of the past image of that block and then applies the archived logs to roll it forward. This technique saves a lot of time by not going to the backups, especially if the backup is on tape.

ZLIB Compression

RMAN offered compression of backup pieces in Oracle Database 10g to conserve network bandwidth but many people were slow to use it. Why? Because third-party compression utilities provided faster alternatives to RMAN’s own. Nevertheless, RMAN 10g compression has some neat features that third-party ones do not provide. For instance, when RMAN 10g restores datafiles, it doesn’t need to uncompress the files first, provided it performed the compression. This approach offers significant bandwidth savings during restores.

In Oracle Database 11g, RMAN offers another algorithm, ZLIB, in addition to the previously available BZIP2. ZLIB is a much faster algorithm but it does not compress a whole lot. On the other hand, it does not consume much CPU either. So if you are CPU starved, it’s a blessing to have ZLIB compression. (Note that BZIP2 is default in version 11.1; you need to license a new option called Advanced Compression Option to use ZLIB.)

To use ZLIB compression, merely set the RMAN configuration parameter:

RMAN> configure compression algorithm 'ZLIB' ;

You have to issue the above command if you have changed it earlier. To change it to BZIP2, issue:

RMAN> configure compression algorithm 'bzip2';

All the compressed backups will use the new algorithm now.

Parallel Backup of the Same Datafile

You probably already know that you can parallelize the backup by declaring more than one channel so that each channel becomes a RMAN session. However, very few realize that each channel can back up only one datafile at a time. So even through there are several channels, each datafile is backed by only one channel, somewhat contrary to the perception that the backup is truly parallel.

In Oracle Database 11g RMAN, the channels can break the datafiles into chunks known as “sections.” You can specify the size of each section. Here’s an example:

RMAN> run {
2>      allocate channel c1 type disk format '/backup1/%U';
3>      allocate channel c2 type disk format '/backup2/%U';
4>      backup
5>      section size 500m
6>      datafile 6;
7> }

This RMAN command allocates two channels and backs up the users’ tablespace in parallel on two channels. Each channel takes a 500MB section of the datafile and backs it up in parallel. This makes backup of large files faster.

When backed up this way, the backups show up as sections as well.

RMAN> list backup of datafile 6;
...
<snipped>
...
    List of Backup Pieces for backup set 901 Copy #1
    BP Key  Pc# Status      Piece Name
    ------- --- ----------- ----------
    2007    1   AVAILABLE   /backup1/9dhk7os1_1_1
    2008    2   AVAILABLE   /backup2/9dhk7os1_1_1
    2009    3   AVAILABLE   /backup1/9dhk7os1_1_3
    2009    3   AVAILABLE   /backup2/9dhk7os1_1_4

Note how the pieces of the backup show up as sections of the file. As each section goes to a different channel, you can define them as different mount points (such as /backup1 and /backup2), you can back them to tape in parallel as well.

However, if the large file #6 resides on only one disk, there is no advantage to using parallel backups. If you section this file, the disk head has to move constantly to address different sections of the file, outweighing the benefits of sectioning.

Backup Committed Undo? Why?

You already know what undo data is used for. When a transaction changes a block, the past image of the block is kept it the undo segments. The data is kept there even if the transaction is committed because some long running query that started before the block is changed can ask for the block that was changed and committed. This query should get the past image of the block—the pre-commit image, not the current one. Therefore undo data is kept undo segments even after the commit. The data is flushed out of the undo segment in course of time, to make room for the newly inserted undo data.

When the RMAN backup runs, it backs up all the data from the undo tablespace. But during recovery, the undo data related to committed transactions are no longer needed, since they are already in the redo log streams, or even in the datafiles (provided the dirty blocks have been cleaned out from buffer and written to the disk) and can be recovered from there. So, why bother backing up the committed undo data?

In Oracle Database 11g, RMAN does the smart thing: it bypasses backing up the committed undo data that is not required in recovery. The uncommitted undo data that is important for recovery is backed up as usual. This reduces the size and time of the backup (and the recovery as well).

In many databases, especially OLTP ones where the transaction are committed more frequently and the undo data stays longer in the undo segments, most of the undo data is actually committed. Thus RMAN has to backup only a few blocks from the undo tablespaces.

The best part is that you needn’t do anything to achieve this optimization; Oracle does it by itself.

Virtual Private Catalog

You are most likely using a catalog database for the RMAN repository. If you are not, you should seriously consider using one. There are several advantages, such as reporting, simpler recovery in case the controlfile is damaged, and so on.

Now comes the next question: How many catalogs? Generally, it makes sense to have only one catalog database as the repository for all databases. However, that might not be a good approach for security. A catalog owner will be able to see all the repositories of all databases. Since each database to be backed up may have a separate DBA, making the catalog visible may not be acceptable.

So, what’s the alternative? Of course, you could create a separate catalog database for each target database, which is probably impractical due to cost considerations. The other option is to create only one database for catalog yet create a virtual catalog for each target database. Virtual catalogs are new in Oracle Database 11g. Let’s see how to create them.

First, you need to create a base catalog that contains all the target databases. The owner is, say, “RMAN”. From the target database, connect to the catalog database as the base user and create the catalog.

$ rman target=/ rcvcat rman/rman@catdb Recovery Manager: Release 11.1.0.6.0 – Production on Sun Sep 9 21:04:14 2007 Copyright (c) 1982, 2007, Oracle. All rights reserved. connected to target database: ODEL11 (DBID=2836429497) connected to recovery catalog database RMAN> create catalog; recovery catalog created RMAN> register database; database registered in recovery catalog starting full resync of recovery catalog full resync complete

This is called the base catalog, owned by the user named “RMAN”. Now, let’s create two additional users who will own the respective virtual catalogs. For simplicity, let’s gives these users the same name as the target database. While still connected as the base catalog owner (RMAN), issue these statement:

RMAN> grant catalog for database odel11 to odel11;

Grant succeeded.

Now connect using the virtual catalog owner (odel11), and issue the statement create virtual catalog

:

$ rman target=/ rcvcat odel11/odel11@catdb

RMAN> create virtual catalog;

found eligible base catalog owned by RMAN
created virtual catalog against base catalog owned by RMAN

Now, register a different database (PRONE3) to the same RMAN repository and create a virtual catalog owner “prone3″ for its namesake database.

RMAN> grant catalog for database prone3 to prone3;

Grant succeeded.

$ rman target=/ rcvcat prone3/prone3@catdb

RMAN> create virtual catalog;

found eligible base catalog owned by RMAN
created virtual catalog against base catalog owned by RMAN

Now, connecting as the base catalog owner (RMAN), if you want to see the databases registered, you will see:

$ rman target=/ rcvcat=rman/rman@catdb

RMAN> list db_unique_name all;

List of Databases
DB Key  DB Name  DB ID            Database Role    Db_unique_name
------- ------- ----------------- ---------------  ------------------
285     PRONE3   1596130080       PRIMARY          PRONE3
1       ODEL11   2836429497       PRIMARY          ODEL11

As expected, it showed both the registered databases. Now, connect as ODEL11 and issue the same command:

$ rman target=/ rcvcat odel11/odel11@catdb

RMAN> list db_unique_name all;

List of Databases
DB Key  DB Name  DB ID            Database Role    Db_unique_name
------- ------- ----------------- ---------------  ------------------
1       ODEL11   2836429497       PRIMARY          ODEL11

Note how only one database was listed, not both. This user (odel11) is allowed to see only one database (ODEL11), and that’s what it sees. You can confirm this by connecting to the catalog as the other owner, PRONE3:

$ rman target=/ rcvcat prone3/prone3@catdb

RMAN> list db_unique_name all;

List of Databases
DB Key  DB Name  DB ID            Database Role    Db_unique_name
------- ------- ----------------- ---------------  ------------------
285     PRONE3   1596130080       PRIMARY          PRONE3

Virtual catalogs allow you to maintain only one database for the RMAN repository catalog yet establish secure boundaries for individual database owners to manage their own virtual repositories. A common catalog database makes administration simpler, reduces costs, and enables the database to be highly available, again, at less cost.

Merging Catalogs

While on the subject of multiple catalogs, let’s consider another issue. Now that you’ve learned how to create virtual catalogs on the same base catalogs, you may see the need to consolidate all these independent repositories into a single one.

One option is to deregister the target databases from their respective catalogs and re-register them to this new central catalog. However, doing so also means losing all those valuable information stored in those repositories. You can, of course, sync the controlfiles and then sync back to the catalog, but that will inflate the controlfile and be impractical.

Oracle Database 11g offers a new feature: merging the catalogs. Actually, it’s importing a catalog from one database to another, or in other words, “moving” catalogs.

Let’s see how it is done. Suppose you want to move the catalog from database CATDB1 to another database called CATDB2. First, you connect to the catalog database CATDB2 (the target):

$ rman target=/ rcvcat rman/rman@catdb2

Recovery Manager: Release 11.1.0.6.0 - Production on Sun Sep 9 23:12:07 2007

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: ODEL11 (DBID=2836429497)
connected to recovery catalog database

If this database already has a catalog owned by the user “RMAN”, then go on to the next step of importing; otherwise, you will need to create the catalog:

RMAN> create catalog;

recovery catalog created

Now, you import from the remote catalog (catdb1):

RMAN> import catalog rman/rman@catdb1;

Starting import catalog at 09-SEP-07
connected to source recovery catalog database
import validation complete
database unregistered from the source recovery catalog
Finished import catalog at 09-SEP-07
starting full resync of recovery catalog
full resync complete

There are several important information in the above output. Note how the target database got de-registered from its original catalog database. Now if you check the database names in this new catalog:

RMAN> list db_unique_name all;

List of Databases
DB Key  DB Name  DB ID            Database Role    Db_unique_name
------- ------- ----------------- ---------------  ------------------
286     PRONE3   1596130080       PRIMARY          PRONE3
2       ODEL11   2836429497       PRIMARY          ODEL11

You will notice that the DB Key has changed. ODEL11 was 1 earlier; it’s 2 now.

The above operations will import the catalogs of all target databases registered to the catalog database. Sometimes you may not want that—rather, you may want to import only one or two databases. Here is a command to do that:

RMAN> import catalog rman/rman@catdb3 db_name = odel11;

Doing so changes the DB Key again.

What if you don’t want to deregister the imported database from the source database during import? In other words, you want to keep the database registered in both catalog databases. You will need to use the “no unregister” clause:

RMAN> import catalog rman/rman@catdb1 db_name = odel11 no unregister;

This will make sure the database ODEL11 is not unregistered from catalog database catdb1 but rather registered in the new catalog.

Back to “Oracle Database 11g: Top Features for DBAs and Developers” homepage